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SUMMARY 


The  Advanced  On-the-job  Training  System  (AOTS)  was  an  Air  Staff  directed, 
AFHRL  developed,  prototype  system  which  designed,  developed,  and  tested  a 
proof-of -concept  prototype  AOTS  within  the  operational  environment  of  selected 
work  centers  at  Bergstrom  AFB,  Texas,  and  Ellington  ANGB,  Texas,  from  August 
1985  through  31  July  1989.  The  AOTS  Readiness  Test  Plan  prescribes  a  series 
of  tests  used  to  verify  that  the  AOTS  software  was  ready  for  installation  in 
the  designated  operational  work  centers  prior  to  the  initiation  of  the  1-year 
System  Level  Test  and  Evaluation  (SLT&E),  which  began  on  1  Aug  1988  and 
concluded  on  31  July  1989.  The  test  prescribes  procedures- to  verify  that  the 
"critical"  functions  are  operating  before  the  software  progresses  to  summative 
evaluations  envisioned  in  the  operational  Active  Duty,  Air  Force  Reserve,  and 
Air  National  Guard  work  centers.  Readiness  testing  was  not  an  exhaustive 
test  of  AOTS  functionality.  Instead,  it  tested  the  small  subset  of  AOTS 
functions  which  were  essential  to  the  usability  of  the  software  in  the  AOTS 
SLT&E  work  centers.  The  AOTS  Readiness  Test  provided  critical  information 
which  was  the  basis  for  the  determination  that  the  system  was  ready  for 
installation  in  the  AOTS  work  centers  and  for  initiation  of  the  1-year  SLT&E. 
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PREFACE 


This  paper  was  developed  by  Ball  Systems  Engineering  Division,  the  AJTS 
on-site  integration  and  management  contractor,  under  Government  Contract 
Number  F33615-C-84-0070.  The  AFHRL  Work  Unit  Number  for  the  project  is 
2557-00-03.  The  primary  office  of  responsibility  for  management  of  the  work 
unit  is  the  Air  Force  Human  Resources  Laboratory,  Training  Systems  Division, 
and  the  Air  Force  AOTS  manager  is  Major  Jack  Blackhurst. 
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APIS  READINESS  IESI.PLAM 


The  essence  of  this  Readiness  plan  can  be  viewed  in  the  form  of  the  attached  graphic,  the  ACTS  Readiness 
Testing  Functional  Flow  Diagram.  This  picture  shows  the  functions  of  the  prototype  AOTS  as  described 
by  the  DAC  Developers  In  the  recently  published  Operational  Guide  to  the  Prototype  AOTS.  The  super¬ 
imposed  boxes  on  the  flow  diegrsm  portray  the  Readiness  tests  as  they  are  currently  envisioned. 

Detailed  preliminary  test  procedures  accompany  the  Flow  Diagram  for  the  Individual  Readiness  Tests. 
Note  that  these  procedures  describe  subsets  of  the  major  functions  that  are  to  be  tested.  The  precise 
layout  of  the  menu  structures  anticipated  in  the  Ada  packages  for  the  critical  functions  Is  required  to 
develop  the  exact  keystroke- by-keystroke  test  procedures. 


1.0  Scope 


1.1  Identification 

The  readiness  test  Is  a  vehicle  to  determine  if  the  software  development  Is  near  Its  promised 
completion.  The  AOTS  development  contract  Is  scheduled  to  enter  Phase  III,  System  Level  Test  and 
Evaluation  (SLT&E),  In  August  1988.  Over  800  Active  Duty,  Air  National  Guard,  and  AF  Reserve 
personnel  will  perform  job  site  training  tasks  and  evaluations  using  the  prototype  AOTS  during  the  one 
year  SLT&E  period. 

The  AOTS  employs  computer  technology  to  develop  and  deliver  Instructional  and  test  materials,  menage 
trainee  progress  through  full  duty  position  qualification,  and  evaluate  the  effectiveness  of  the  Air  Force 
Job  site  training  system.  This  document  describes  a  series  of  tests  to  verify  that  the  AOTS  software  Is 
ready  for  Installation  In  the  designated  operational  workcenters. 

1.2  Introduction 

1.2.1  Background 

The  AF  Human  Resources  Laboratory  Issued  a  contract  In  1 985  to  the  Douglas  Aircraft  Company  ( DAC) 
to  design  and  build  a  prototype  On-the-Job  Training  (OJT)  system  using  state  of  the  art  computer  aided 
instructional  system  techniques.  The  Advanced  On-The-Job  Training  System  (AOTS)  encompasses  as¬ 
pects  of  Instructional  systems  development,  principles  of  learning,  computer  aided  courseware  devel¬ 
opment,  Interactive  training,  automated  scheduling,  and  testing. 

Software  was  developed  in  two  phases.  Technical  progress  during  the  first  phase,  Preliminary  Design, 
was  reported  to  the  Air  Force  by  face-to-face  meetings  at  preliminary  design  reviews.  These  design 
reviews  were  documented  by  Critical  Item  Specifications,  Prime  Item  Specifications,  and  finally,  by 
Air  Force  acceptance  of  the  Systems  Level  Specification. 

The  second  phase,  Detailed  Design,  is  scheduled  to  be  completed  by  31  July  1988  with  the  delivery  to 
the  Air  Force  of  the  completed  Advanced  On-the-Job  Training  System.  The  completeness  and  function¬ 
ality  of  the  entire  system  cannot  be  measured  until  this  time. 

To  perform  the  AOTS  functions,  the  following  data  elements  will  be  generated  by  the  various  AOTS 
components: 


1 .2. 1 . 1  Automated  Airman  Training  Records  ( ATRs):  personnel  data,  status  of 
ongoing  training,  and  training  completion  ( including  retraining,  cross  training,  upgrade 
training,  etc.)  histories.  , 


f  .2. 1 .2  Mttstsr  Task  Lists  ( MTLs).  an  all  inclusive  list  of  job  related  tasks 
assigned  to  each  Air  Force  Speciality  (AF5). 

1 .2. 1 .3  Behavioral  Objectives:  terminal  and  supporting  behavioral  objectives  to 
support  job  site  training  and  evaluation  of  tasks. 

1 .2. 1 .4  Evaluation  Materials:  knowledge  and  performance  tests  for  use  in 
determining  attainment  of  behavioral  objectives. 

1 .2. 1 .5  Other  Training  Requirements:  ancillary  training,  additional  duty 
'  training,  contingency  tasks,  and  career  development  courses. 

1 .2. !  .6  Event  Schedulers:  training  and  evaluation  event  schedules,  training 
resource  requirements. 

1 .2. !  .7  System  Evaluation  data:  test  results,  training  times,  airmen 
qualifications,  and  other  data  used  to  indicate  training  efficiency  and  effectiveness. 

1.2.2  Purpose 

Determine  if  software  is  ready  for  SLT&E  in  an  operational  environment 


1 .3  Relationship  to  Current  Testing 

The  ACTS  is  undergoing  the  first  of  two  general  classifications  of  testing:  Formative  and  Summatlve 
tests.  Formative  evaluation  activities  are  conducted  internally  by  the  program  developers  to  determine 
the  degree  of  attainment  of  specific  program  goals  and  to  pinpoint  subcomponent  performance  goals  not 
yet  attained  Summatlve  evaluation  Is  directed  to  a  more  general  assessment  of  the  degree  to  which 
broader  program  outcomes  have  been  attained. 

The  Formative  test  end  evaluation  of  AOTS  during  Phase  1 1  has  been  primarily  concerned  with  verifying 
accomplishment  of  technical  performance  specifications  for  subcomponents,  components,  interfaces, 
and  subsystems.  Formative  evaluation  results  were  used  to  revise  or  modify  the  system  elements  to 
achieve  at  or  above  acceptable  levels.  In  the  upcoming  Summatlve  evaluation,  the  entire  system  will  be 
assessed  against  four  critical  issues:  1 )  Compliance  with  system  specifications;  2)  Performance  at  or 
above  specified  levels;  3)  Suitability  for  correcting  deficiencies  In  the  existing  operational  job-site 
training  arena;  and  4)  Acceptance  by  the  users. 

The  transition  from  Formative  to  Summatlve  evaluation  will  involve  changes  in  both  primary  users 
and  uses  of  the  system.  The  development  of  the  "tools"  necessary  to  conduct  OUT  and  OJT  management 
were  the  focus  in  Formative  testing.  During  Formative  testing,  members  of  the  Instructional  Systems 
Team  ( 1ST)  were  the  key  AOTS  users  and  performed  primary  functions  such  as  Master  Task  List  devel¬ 
opment,  Performance  and  Evaluation  criteria  establishment,  and  authoring  of  test  items,  in  Summa- 
tive  evaluation,  the  primary  users  will  be  the  airmen  In  the  operational  work  centers  and  the  key  uses 
will  transition  to  those  functions  related  to  conducting  on-line  job  knowledge  skills  Improvements  and 
evaluations.  The  success  of  Summatlve  evaluation  is  dependent  on  the  AOTS  software  being  fully  opera¬ 
tional  when  the  system  Is  Installed  in  the  workcenters.  Hence,  there  is  a  need  to  determine  the 
"readiness"  of  the  AOTS  to  enter  Summatlve  evaluation,  or  Systems  Level  Test  &  Evaluation  (SLT&E). 

This  Readiness  Test  Plan  prescribes  procedures  to  verify  that  the  "critical"  functions  are  operating  be¬ 
fore  the  software  progresses  to  Summatlve  evaluations  envisioned  In  the  operational  Active  Duty,  Air 
Force  Reserve,  and  Air  National  Guard  Workcenters.  Readiness  testing  will  not  be  an  exhaustive  test  of 
AOTS  functionality.  Instead,  It  will  test  the  small  subset  of  AOTS  functions  which  are  essential  to  the 


usability  of  the  software  in  the  SLT&E  wcrkcenter*. 


2.0  References 


The  following  references  were  used  to  generate  the  sections  of  this  Readiness  Test  Plan: 

1 )  AOTS  Statement  of  Work  (SOW) 

Section  C  -  Description/Specifications 
Date:  16  April  1984 

2)  ACTS  System  Specification 

Spec  No:  70S647000 

Cods  Ident  No:  76301 

Date: :  5  May  1986 

3)  AOTS  Management  Prims  Stem  Development  Specification  (81) 

Spec,  No:  70S647100 

Code'ldent.  No:  76301 

Date:  13  March  1987 

4)  AOTS  Management  Computer  Program  Configuration  item  Development 
Specification  (85) 

Spec.  No:  70S64741 1 

Codeldent.No:  7630! 

Date:  7  April  1986 

5)  AOTS  Evaluation  Prime  Item  Development  Specification  (8!) 

Spec.  No:  70S647300 

Code  Ident  No:  7630! 

Date:  28  March  1986 

6)  AOTS  Evaluation  Computer  Program  Configuration  Item  Development 
Specification  (85) 

Spec.  No:  70S647413 

Code  Ident  No:  7630! 

Date:  18  April  1986 

t 

7)  BSED/BDM  AOTS  Requirements  Traceability  Matrix 

Date:  15  April  1988 

8)  AFM  55-43  Volumes  I  and  II,  "Management  of  Operational  Test  and  Evaluation* 

Date:  13  June  1979 


3.0  Test  Pleasing  Assumptions  &  Constraints 


3.1  Assumptions  Hade  In  last  Planning 

As  discussed  above,  the  AOTS  Readiness  Test  will  not  be  an  exhaustive  test  of  AOTS  functionality. 

instead.  Readiness  Testing  will  focus  on  a  set  of  functions  which  wars  judged  to  be  critical  to  the  success  of 

the  AOTS  SLT&E.  The  selection  of  critical  functions  was  based  on  the  following  assumptions: 

3.1.1  AOTS  functions  can  be  divided  Into  two  classes;  user  functions,  such  as  managing  Airman 
Training  Records,  and  developer  functions  such  as  cresting  training  modules  and  evaluation  instru¬ 
ments.  SLT&E  will  take  place  in  the  workcenter/user  environment.  The. -afore,  Readiness  Testing 
should  focus  on  user  .functions, 

3.1.2  The  Readiness  Test  is  a  software  test  Th8  Personnel  and  Support  Subsystem  does  not  include 
software  and  will  therefore  not  be  addressed  by  the  Readiness  Test. 

3. 1 .3  AOTS  includes  four  Computer  Program  Configuration  Items  (CPCIs);  Management,  Evalua¬ 
tion,  Training  Development  and  Delivery  ( TD&D),  and  System  Support.  The  TD&D  CPCI  was  a  separ¬ 
ate  development  effort.  TD&D  software  Is  taken  from  ISS  which  Is  not  directly  maintained  by  the 
AOTS  developers.  Functons  performed  by  the  TD&D  CPCI  will  therefore  not  be  included  in  Readiness 
Testing. 

3. 1 A  The  System  Support  CPCI  consists  of  >ow  level  software  which  will  not  be  directly  visible  to 
the  AOTS  workcent9T  users.  Readiness  Testing  will  address  only  the  Access  Control  function  of  the 
System  Support  CPCI.  Because  access  control  safeguards  the  integrity  of  the  personnel  data  In  the 
AOTS  database.  It  is  Judged  to  be  critical  to  SIT&E  success. 

As  a  result  of  assumptions  1  through  4,  the  Readiness  Test  will  focus  on  user  functions  of  the  Management  and 
-Evaluation  CPCIs. 

Some  additional  assumptions  which  are  fundamental  to  the  AOTS  Readiness  Test  Planning  ere 

/ 

3. !  .5  The  purpose  Gf  Readiness  Testing  is  to  decide  whether  to  proceed  with  AOTS  SLT&E  based  on 
the  usability  of  the  AOTS  software.  This  decision  rests  with  AFHRL;  therefore,  AFHRL  will  approve 
the  AOTS  Readiness  Test  Plan  and  perform  the  Readiness  Tests. 

3.J.6  The  AOTS  software  will  be  under  formal  Configuration  Control  prior  to  the  start  os  Readiness 
Testing.  Problems  uncovered  during  the  Readiness  Test  will  be  documented  as  Software  Problem  Re¬ 
ports  (SPRs)  and  resolved  through  normal  configuration  management  procedures  for  hitfi  priority 
SPRs. 


3.2  Conditions  Required  During  Testing 

In  order  to  Implement  an  effective  Readiness  Test,  the  following  conditions  must  be  met: 

1 .  A  separate  Readiness  Test  environment  must  be  established  on  the  VAX  at  Brooks. 

2.  A  complete  copy  of  the  AOTS  software  must  be  Installed  in  the  Readiness  Test  environment.  This 
software  should  be  the  same  version  which  will  be  used  for  SLT&E,  and  must  be  under  version 
control. 

3.  Sample  AOTS  databases  must  be  established  In  the  Readiness  Test  environment.  The  data  should  be 
representative  of  SLT&E  data,  but  must  be  in  a  separate  environment  to  prevent  contamination  or 
alteration  of  production/SLT&E  data. 


/ 


4.0  ACTS  Reef!  mm  TMt  Plan  . 

4.!  AOTS  Function*  To  Be  Tasted 
4.1.1  Critical  Functions 

Meetings  between  personnel  from  the  AFHRL ,  DAC  aid  BSED  In  March,  1 988  led  to  the  compilation  of  a 
list  of  functions  critical  to  the  implementation  of  the  AOTS  Into  the  operational  SLT&E  environment. 
The  AOTS  functions  determined  to  be  critical  included: 

1)  Access  control 

2)  Interfacing  with  the  ITR 

3)  Scheduling  training  and  evaluation  events 

4)  Accessing  the  MTL 

5)  Requesting  evaluation  instruments 

6)  Controlling  off-line  evaluation  Instruments 

7)  Updating  ATRs 

* 

The  AOTS  critical  functions  are  described  in  soma  detail  in  Paragraph  4.2. 1 


4. 1. 1.1  Rationale 

Before  the  AOTS  is  Installed  in  operational  workcenters  at  Bergstrom  AFB  and  Ellington  ANGB ,  the 
critical  functions  listed  above  must  operate  correctly.  If  they  do  not,  the  AOTS  will  be  unusable  for  test 
subjects  In  the  workcenters.  If  the  AOTS  critical  functions  do  not  work  properly,  the  requirement  that 
the  test  workcenters  use  the  AOTS  may  significantly  Interfere  with  performance  of  mission  critical 
functions. 

*•  The  Readiness  test  effort  will  focus  largely  upon  the  Management  and  Evaluation  Subsystem  functions. 
Within  these  subsystems  are  the  functions  which  will  be  accessed  most  frequently  by  workcenter  per¬ 
sonnel  early  in  the  SLT&E.  Two  notable  functions  which  will  not  be  tested  are: 

/ 

1)  QC  evaluations 

2)  Generating  reports 

These  functions  are  important  to  the  operation  of  the  overall  AOTS,  however,  they  are  not  critical  to  the 
use  of  the  system  during  the  early  stages  of  SLT&E. 

4.2  AOTS  READINESS  TEST  DESIGN 

The  AOTS  Readiness  Test  design  was  derived  In  general  from  the  test  planning  guidance  contained  In  Air 
Force  Manual  (AFM)  55-43,  Management  of  Operational  Test  ami  Evaluation.  AFM  55-43  suggests  a 
test  design  which  is  sat  up  according  to  the  following  logical  flow. 

CRITICAL  QUESTION  —  >  GENERAL  TEST  OBJECTIVE  —  >  SPECIFIC  TEST  OBJECT  1VE(S)  —  >  MEASURE 
(S)  OF  EFFECTIVENESS  — >  EVALUATION  CRITERIA 

The  first  step  is  to  postulate  a  general  question.  For  example,  the  first  Reediness  Test  critical  question 
is: 

-  Can  the  AOTS  generate  an  ATR? 

Then  a  gsnarai  test  objective  is  formulated  begad  on  the  critical  question: 

-  Evaluate  the  AOTS  in  gensrsting  an  ATft. 


/ 


From  the  general  test  objective  come  one  or  more  specific  test  objectives: 

1 .  Determine  whether  the  ATR  Is  initialized  with  PDS  data. 

2.  Determine  whether  the  AOTS  provides  for  manual  entry  of  PDS  data 

3.  Determine  whether  the  AOTS  provides  for  manual  entry  of  completions  Into  an 
airman's  ATR. 

To  determine  If  a  specific  test  objective  has  been  met,  a  "measure  of  effectiveness"  ( MOE)  must  be  de¬ 
termined.  In  this  series  of  Readiness  Tests,  all  MOEs  will  be  observations  of  whether  or  not  the  func¬ 
tion  being  assessed  worked.  Continuing  with  the  above  example,  the  following  MOEs  are  derived  from 
the  specific  test  objectives: 

1 .  Observe  an  attempt  to  Initialize  ATR  with  PDS  data 

2.  Observe  an  attempt  to  manually  enter  PDS  data  into  an  ATR. 

3.  Observe  an  attempt  to  manually  enter  completions  into  an  ATR. 

Finally.  eech;MOE  will  have  a  particular  evaluation  criteria 

1.  Performed  Initialization/Did  not  perform  initialization. 

2.  Allowed  entry/Did  not  allow  entry. 

3.  Allowed  entry/Did  not  allow  entry. 

This  test  design  logic  has  led  to  the  development  of  fourteen  specific  "Readiness  Tests".  Each  of  these 
tests  is  specifically  related  to  one  or  more  of  the  seven  AOTS  critical  functions.  The  AOTS  Readiness 
Tests  are  listed  below: 

Test  f  Trainee  access  control 

Test  2  Trainer  access  control 

Tast3  Evaluator  access  control 

Test  A  Supervisor  acoess  control  , 

Tests  Training  Manager  access  control 
Test  6  Generate  ATR 
Test?  Generate OPTR 
Tests  Generate ITR 

Test  9  ITR  Training  Requirement  Rank  Ordering 

Test  10  Schedule  Training  —  1  Airman 

Test  1 1  Schedule  Training  —  Many  Airmen 

Test  12  Generate  Evaluation  Schedule 

Test  13  Generate  On-line  Test 

Test  M  Generate  Off-line  Test 

TestlS  Generate  Off-line  PEC 

Test  16  Score  On-line  Test 

Test  17  Score  Off-line  Symbolic  Test 

Test  18  Score  Off-line  PEC 

Test  19  Record  Performance 


READINESS  TEST/ CRITICAL  FUNCTION  MATRIX 
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Table  4-7 

4.2. 1  The  rationale  for  the  Readiness  Test/AOTS  critical  function  relationships  are  explained 
in  the  following  paragraphs. 

4.2. 1 . 1  Access:  Control  To  perform  any  testing  on  the  AOTS  the  tester  must  access  specific 
AOTS  packages.  Access  control  will  be  specifically  tested  In  the  Access  Control  Test  ( Test  1 5).  This 
test  will  involve  validating  proper  access  by  each  critical  user  type  (Trainee,  Trainer,  Supervisor, 
Evaluator,  Training  Manager)  into  only  properly  authorized  AOTS  components.  This  critical  function 
is  therefore  related  to  each  of  the  AOTS  Readiness  Tests. 

4.2. 1 .2  Interface  With  ITR  The  critical  functlqn  of  Interface  with  the  ITR  Involves  a  va¬ 
riety  of  interaction  with  the  ITR  editor  and  existing  ITRs.  The  user  must  be  first  able  to  generate  an 
Initial  ITR.  If  the  requirements  listed  in  the  ITR  are  not  in  the  desired  order,  the  user  must  be  able  to 
modify  the  order.  The  user  must  be  able  to  access  a  listing  of  ITR  requirements  to  schedule  an  airman 
for  training.  Finally ,  the  ITR  must  be  accessible  for  automatic  update  of  training/evaluation/ 
completion. 

4.2. 1.3  Schedule  Training  and  Evaluation  Events  The  critical  function  of  scheduling 
training  and  evaluation  events  involves  three  types  of  scheduling  functions:  First  the  scheduling  of 
training  for  Individual  airmen,  Secondly  the  scheduling  of  training  for  multiple  airmen,  and  thirdly 
the  generation  of  evaluation  schedules.  As  mentioned  In  the  paragraph  above,  the  user  must  have  access 
to  a  listing  of  ITR  requirements  In  order  to  schedule  airmen  for  required  training. 

4.2. 1.4  MTL  Access  The  critical  function  of  MTLacoess  Involves  a  limited  MTL  access.  The 
MTL  must  be  accessible  when  generating  or  modifying  an  OPTR  so  that  desired  MTL  tasks  can  be  copied 
Into  an  OPTR.  The  MTL  must  also  be  accessible  when  generating  or  modifying  an  ITR  so  that  tasks  can 
be  copied  into  an  ITR. 

4.2. 1 .5  Request  Evaluation  Instrument  Evaluation  instruments  will  be  requested  when 
users  initially  generate  on  or  off-line  evaluations.  The  request  for  Instruments  must  also  work  ac¬ 
ceptably  for  the  scoring  of  evaluations. 


4.2. 1.6  Control  Off-Line  Evaluation  Instrument  The  control  of  off-line  evaluation 
instruments  occurs  from  the  time  that  an  evaluation  instrument  is  generated  off-line  through  the  time 
that  it  is  scored  and  then  destroyed.  AOTS  control  functions  occur  at  the  point  of  generation  and  the 
point  of  scoring.  As  indicated  In  Table  4- 1 ,  there  are  three  tests  within  the  Readiness  Testing  for 
which  this  critical  function  will  be  tested. 

4.2. 1 .7  Update  ATR  The  critical  function  of  ATR  update  includes  both  the  initial  creation  of 
the  ATR  and  the  update  of  the  ATR  with  trainee  training  and  evaluation  completions.  The  AOTS  Readiness 
Test  will  test  this  critical  function  in  both  of  these  areas. 

4.2.1  Test  Objectives  (Figure  4-1,  p.  21) 

Figure  4-  i ,  entitled  “AOTS  Readiness  Test  Functional  Flow  Diagram",  details  the  flow  of  functional 
processes  that  will  occur  during  the  initial  operation  of  the  AOTS  in  the  workcenters.  The  fine  line 
boxes  on  the  diagram  show  specific  AOTS  functions  which  are  included  in  each  Readiness  test.  Each  test 
is  designed  to  address  one  or  mere  specific  test  objectlvef  s)  as  desribed  below. 

4.2. 1. 1  Test  Objective  1  -  Eveluate  The  AOTS  In  Providing  Access  Control  To 
Trainees 

in  testing  the  critical  function  of  access  control ,  it  is  necessary  to  test  the  specific  access  provided  to 
trainee  users  of  the  system.  Trainees  will  need  access  to  listings  of  personal  training  requirements, ; 
all  AFS  tasks  which  have  been  certified,  other  training  accomplishments,  and  status  of  all  training  un¬ 
derway.  Trainees  also  need  access  to  materials  that  are  available  for  knowledge  training  and  testing. 

4.2. 1.1.1  Test  Objective  1  Design  (Figure  4-2) 
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4.2. 1 .2  Test  Objective  2  -  Evaluate  The  AOTS  in  Providing  Access  Controls 
To  Trainers 

The  Testing  of  access  control  also  involves  examining  the  controls  and  prlvillges  provided  to  trainer 
users  of  the  system.  Trainers  need  to  be  able  to  a)  review  on-line  records  of  trainees  to  determine 
training  requirements  and  the  status  of  progress;  b)  review  the  task  records  to  determine  the  sub- 
tasks,  activities,  supporting  knowledge  and  skills  and  other  task  elements,  the  sequences  In  which 
training  must  be  accomplished,  end  the  resources  required  for  task  performance;  and  c)  schedule 
training  events. 


4.2. 1.2.1  Test  Objective  2  Design  (Figure  4-3) 


_ 1 

critical  question 

SPGCIfIC  TEST  OOJSCTIWS 

EVALUATION  CRITERIA'  '< 

CWOttAOTSpre** 
CflMKMlCtlM  bi 

tTMMn? 

CvMMUUnMTS 

CatMTMtU  tTMMn. 

1)  OttsnMM  trtwUMr 

tnMwr*  art  Ml*  U  act*** 
Uwtr  trtMM't  «HIm 
trwmnf  rtetrot. 

i)  ON  tnw  ettaawa  ay  treNer  te 
H6IM  tTMAMT* aft-l(M 
nicerli. 

1)  Marti  aaurt/ 
RscanNnet 

KdUl^, 

it  otfrmtm  wmumt  trMMn 
«n  am  u  kcwi  AOTS 
tea*  IMtlMf*. 

2)  Owrw  tttmn  trainer 
te  access  MJT%  tack 

IICtMf*. 

2)  Hatleys  accesses/ 
Llsltnfi  Mt 
awart. 

J)  CMltrmM*  «MUw  traloera 
era  Ml  la  ecceea  ACTS 

ruKtlOM. 

3)  Wwnw  attentat  kg  trainer 

U  ecce «*  AOTS  MM)ll«t 
rmettene. 

3)  Functtenc  atwmi 
Functions  rat 

«CHat  a 

4.2. 1.3  Test  Objective  3  -  Evaluate  The  AOTS  in  Providing  Access  Controls  to 
Evaluators 

Access  control  testing  involves  testing  of  access  provided  by  the  AOTS  to  Evaluators.  Evaluators  should 
have  all  of  the  accesses  given  to  Trainers.  Additionally,  Evaluators  need  access  to  evaluation  materials 

to  be  used  off- line. 

> 

4.2. 1.3. 1  Test  Objective  3  Design  (Figure  4-4) 


4.2. 1.4  Test  Objective  4  -  Evaluate  The  AOTS  in  Providing  Access  Controls  to 
Supervisors 

The  testing  of  access  control  includes  examintatton  of  accesses  and  privileges  provided  to  workcenter 
supervisors.  Supervisors  need  the  capability  to  a)revfew  the  training  records  of  all  personnel  they 
supervise;  b)  schedule  training  and  evaluation  events;  c)  prioritize  training  requirements;  d)  review 
task  records;  e)  obtain  test  materials  for  off-line  use;  and  f)  certify  trainee  qualifications. 


4.2. 1.4.1  Test  Objective  4  Design  (Figure  4-5) 
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4.2. 1 .5  Test  Objective  5  -  Evaluate  The  AOTS  In  Providing  Access  Controls  to 
Training  Managers 

Access  Control  testing  Includes  testing  of  accesses  provided  to  training  managers. 

4.2. 1 .5. 1  Test  Objective  5  Design  (Figure  4-6) 


4.2.1. 1  Test  Objective  6  -  Evaluate  The  AOTS  In  Generating  An  ATR 

In  testing  the  critical  function  of  ATR  update,  It  Is  necessary  to  assess  the  capability  to  Initially  gener¬ 
ate  an  ATR.  The  ATR  Is  a  historical  account  of  all  prior  training  received  by  an  Individual  airman.  This 
data  Is  required  for  workoenter  management  personnel  to  determine  the  specific  training  required  for 
airmen  In  the  workcenter  (generation  of  Individual  Training  Requirements  ( ITRs)).  This  will  be  a 
fundamental  step  In  the  operation  of  the  AOTS  In  the  workcenters. 


4.2.  M.  1  Test  Oojeetive  6  Design  (Figure  4-7} 
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4.2. 1.2  Test  Objective  7  -  Evaluate  the  AOTS  In  Generating  An  OPTR 

In  order  to  test  the  critical  functions  of  MTL  access,  ITR  interface,  and  scheduling  training  and  evalua¬ 
tion  events,  AOTS  must  be  capable  of  generating  an  OPTR.  The  first  step  in  generating  an  OPTR  is  to 
select  a  Generic  Position  Task  Requirements  (GPTR)  list  for  the  type  of  duty  position  to  which  the  air¬ 
man  has  been  assigned,  GPTRs  identify  tasks  performed  by  all  persons  assigned  to  equivalent.duty  posi¬ 
tions  throughout  the  Air  Force.  For  the  prototype  AOTS,  the  GPTRs  have  been  (or  are  being)  input  by 
the  subject  matter  experts  ( SMEs)  assigned  to  this  R&D  effort.  The  immediate  supervisor  uses  the  ap¬ 
plicable  GPTR  as  the  baseline  to  define  specific  training  requirements  for  that  particular  duty  position 
-  thus,  the  OPTR  is  generated.  The  supervisor  can  add  tasks  from  the  appropriate  Master  Task  List 
( MTL),  other  training  requirements  ( ie. ,  ancilery  training),  or  local  task  requirements. 


4- 

4.2. 1.2.1  Test  Objective  7  Design  (Figure  4-8) 
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4.2.  f  .3  Test  Objective  8-  Evaluate  The  ACTS  In  Oeneratlng  an  ITR 

Before  it  Is  possible  to  examine  the  critical  function  of  ITR  Interface,  the  capability  of  the  AOTS  to  gen¬ 
erate  an  Initial  ITR  must  be  tested.  The  generation  of  an  airman’s  ITR  provides  a  list  of  training  re¬ 
quired  for  the  attainment  of  position  qualification.  This  list  of  training  requirements  Is  necessary  to 
schedule  the  airman  for  required  training  and  evaluation  events. 

4.2. 1.3. 1  Test  Objective  8  Design  (Figure  4-9) 
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4.2. 1.4  Test  Objective  9-  Evaluate  the  AOTS  In  Automatically  Rank  Ordering 
ITR  Training  Requirements 

An  additional  aspect  of  the  critical  function  of  ITR  interface  Is  the  automatic  rank  ordering  of  ITR 
training  requirements  by  the  AOTS.  This  capability  is  required  to  assess  the  critical  function  of  sched¬ 
uling.  For  trainees  and  supervisors  to  understand  the  most  efficient  order  In  which  to  receive/deliver 
training,  the  AOTS  must  list  the  training  requirements  in  order  of  priority.  It  is  also  Important  for 
workcenter  management  personnel  to  have  the  capability  to  override  the  automatic  rank  ordering  func¬ 
tion  when  mission  requirements  necessitate  a  different  training  order. 

4.2.1.4.1  Teat  Objective  9  Design  (Figure  4-10) 
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4.2. 1.5  Test  Objective  10-  Evaluate  the  AOTS  Capabilities  for  Scheduling 
Training  for  an  Individual  Airman 

The  critical  function  of  scheduling  training  and  evaluation  events  Involves  the  scheduling  of  training  for 
either  one  or  many  airmen.  The  AOTS  Individual  airman  scheduling  function  provides  workcenter  man¬ 
agement  personnel  with  two  fundamental  capabilities.  First  the  manager  Is  able  to  identify  specific 
training  tasks,  times  and  trainers  for  Individual  trainees.  Secondly,  the  manager  is  able  to  identify 
whether  the  trainee  has  been  scheduled  for  any  conflicting  training  or  evaluation  events. 


4.2. 1 .5. 1  Test  Objective  1 0  (Figure  4-11) 
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4.2. 1.6  Test  Objective  1 1-  Evaluate  the  AOTS  Capabilities  for  Scheduling 
Many  Airmen 

In  addition  to  the  above  objective  regarding  the  AOTS  capability  to  schedule  an  individual  airman,  an  ob¬ 
jective  is  required  to  test  the  capability  of  the  AOTS  to  schedule  many  airmen.  This  function  allows 
workcenter  management  personnel  to  specify  particular  training  requi rement(  s) ,  and  to  then  receive  a 
«.  list  of  all  airmen  requiring  training  on  that  requirement.  As  with  the  individual  airman  scheduling 

function,  the  "many  airmen"  scheduling  function  provides  for  the  specification  of  training  date/time/ 
location,  etc.  .  and  for  the  Identification  of  any  schedule  conflicts. 

4.2. 1 .6. !  Test  Objective  1 1  Design  (  Figure  4- 1 2) 
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4.2. 1 .7  Test  Objective  1 2-  Evaluate  the  ACTS  In  Generating  An  Evaluation 
Schedule 

An  additional  test  of  the  critical  function  of  scheduling  training  and  evaluation  events  Involves  exami¬ 
nation  of  the  ACTS  capability  in  generating  evaluation  schedules.  This  function  will  provide  capability 
for  workcenter  personnel  to  create  time-ordered  listings  of  evaluation  events. 

4.2. 1.7.1  Test  Objectives  12  Design  (Figure  4-13) 
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4.2. 1 .0  Test  Objective  8  3-  Evaluate  the  AOTS  Capability  To  Generate  An 
On-Line  Test 

The  critical  function  regarding  request  of  evaluation  instruments  Includes  generation  of  on-line  evalu¬ 
ations.  This  objective  provides  for  the  testing  of  ACTS  capability  in  this  area.  The  on-line  test  gsnera- 
tion  capability  allows  workcenter  personnel  to  view  on-screpn  evaluations  at  the  workcenter  termi¬ 
nals.  This  provides  access  to  computer-assisted  evaluations  in  the  workcenters. 


4.2. 1 .0. 1  Tost  Objective  8 3  Design  (Figure  4- 1 4) 
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4.2. 1 .9  Test  Ofejwtlve  1 4-  Evaluate  The  AOTS  Capability  To  Generate  Off- 
Line  Evaluations 

Testing  AOTS  capability  to  generate  off-line  evaluations  is  an  additional  examination  of  the  requesting 
evaluation  instrument,  and  the  control  of  off-line  evaluation  instruments  critical  functions.  The  off¬ 
line  evaluation  generation  capability  allows  personnel  to  generate  hard  copies  of  evaluations  at  work- 
center  printers.  Workcenter  personnel  can  automatically  generate  "AOTS  scorable"  tests  for  trainee 
evaluation  purposes.  The  capability  brings  with  It,  however,  the  problem  of  controlling  the  hard  cop¬ 
ies  of  evaluations  that  are  generated.  The  AOTS  generated  control  procedures  will  also  be  tested  under 
this  objective. 
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4.2. 1. 5,1  Test  Objective  14  Design  (Figure  4-15) 


OENKAL  TOT  OBJECTIVE 

Metric  Ttsr  objictivcs 

MEASURES  Of  CffECTIVENESS 

(VALUATION  CRITERIA 

Cm  tlw  AOTS 

ganarata 

ofMtna  auotaatl  anaf 

(MtwuttoAors 

CMMtUty  UftMTlU 

t)  Datarmfna  wnatna*  ACTS 
m tract*  afr-tlaa  aymtallc 
tMlItOIM. 

I)  OSaarvt  attampt  to  aatroct 
taat  Itama. 

1)  Extract  itama/ 

Old  nat  axtract  Itama. 

2)  Oatarmlaa  wNatAar  ACTS 
will  nmnl»  m  aff-llna 
aymoallc  taat 

2)  OPaarvo  attarcpt  ta  ganarata 
taat. 

2)  Sanarotad  taat/ 

Old  not  ganareta  taat 

5)  Ootarmtoa  wAatftar  AOTS 
lagathaatatuoef 
ganarataa  alf-llna 
agwoalic  taat*. 

5)  Otaarva  AOTS  accuracy  of 
atatua  log. 

3)  Accuratalog/ 

Inaccurtta  log. 

4.2. 1.10  Test  Objective  1 5-  Evaluate  The  AOTS  Capability  To  Generate  An 
Off-Line  Performance  Evaluation  Checklist  (PEC) 

In  addition  to,  off-line  evaluations,  th8  AOTS  will  generate  hard  copies  of  PECs  at  workcenter  printers. 
Off-Line  PEC: generation  Is  an  additional  aspect  of  the  request  evaluation  Instrument  and  control  of  off¬ 
line  evaluation  instruments  critical  functions. 

,4.2.1.10.1  Test  Objective  15  Design  (Figure  4-16) 
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4.2. 1.11  Test  Objective  1 6-  Evaluate  the  AOTS  in  Scoring  An  On-Line 
Test 

The  scoring  of  on-line  tests  Is  related  to  several  of  the  critical  functions.  For  example  the  ITR  could 
not  be  updated  with  on-line  training  completions  If  the  on-line  evaluations  were  not  scored.  It  Is  re¬ 
lated  to  the  Update  ATR  critical  function  because,  as  with  the  ITR,  the  ATR  cannot  be  updated  with  on¬ 
line  training  completions  If  the  on-line  evaluations  are  not  graded. 


4.2.1. 1 1.1  Test  Objective  16  Design  (Figure  4-17) 
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4.2. 1.12  Test  Objective  1 7-  Evaluate  the  A0T5  In  Scoring  An  Off-Line 
Symbolic  Test 

The  earing  of  off-line  symbolic  tests  Is  related  to  the  seme  critical  functions  listed  above  for  on-line 
tests.  In  addition,  this  test  objective  is  related  to  the  critical  function  of  controlling  off-line  evaluation 
Instruments.  The  process  for  scoring  the  off-line  test  involves  procedures  for  controlling  the  instru¬ 
ment. 

4.2.1.12.1  Test  Objective  17  Design  (Figure  4-18) 


atmcAL  oxstiom 

■NOUL  TEST  OBJECTIVE 

STOJEIC  TEST  OBJECTIVES 

ICAMJRCS  or  ITrtCTIVEJCSS 

evaluate*  carrwiA 

Cm  tka  ACTS  (c*r» 
m  *fr-tww 
mmm He  t«*t? 

Evaluate  tka  AOTS 
laaearia|«»eff-«ke 
•gmMMctmt 

Egg 

1)  ONmt  attemat  ta  race  tact 

tfluora. 

1)  Tact  waa  read/ 
TdtKWWlrM. 

EggS 

2)  MWW  Ittwst  AOTS 

U  inert!  tact. 

2)  TwIwwwlW/ 
TMmiMMaM 

- 

jgfefl 

3)  Okaarve  attaaapt  ta  acara 
teat. 

3)  Taat  waa  acarad/ 

Tact  waa  kat  acarad. 

t 

BBi 

4)  Okaarva  attempt  accaaa  mar*. 

4)  Taat  waa  aoeaaatMa/ 

Twl  warn  imccewlMe. 

4.2. 1.13  Test  Objective  18-  Evaluate  the  AOTS  In  Scaring  An  Off-Line 
Performance  Evaluation  Checklist 

The  test  objective  for  scoring  off-line  PECs  also  includes  all  of  the  critical  function  relations  listed  for 
scoring  on-line  test  and  those  listed  above  for  scoring  off-line  symbolic  tests. 

4.2. 1 . 1 3. 1  Test  Objective  1 8  Design  ( Figure  4- 1 9) 
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4.2.1.14  Test  Objective  19-  Evaluate  The  AOTS  In  recording  Trainee 
Performance 

Evaluating  the  AOTS  in  recording  trainee  performance  relates  to  the  critical  function  of  interfacing 
with  the  ITR  and  updating  the  ATR.  This  relationship  Is  based  on  the  fact  that  when  trainee  performance 
is  recorded  It  Is  recorded  In  both  the  trainee’s  ITR  and  ATR. 


/ 


4.2.1.14.1  Test  Objective  19  Design  (Figure  4-20) 
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4.2.2  Pass/Fall  Criteria 

for  each  specific  test  objective  described  In  Section  4. 1 ,  an  MOE  was  defined  that  provided  a  straight 
forward  "go/no  go"  result.  It  is  not  likely  that  every  specific  objective  and  the  associated  MOE  must 
achieve  a  "go"  observation  for  the  AOTS  to  pass  the  Readiness  Test.  In  Instances  where  there  are  more 
than  one  specific  test  objective  for  a  given  general  objective,  a  "go"  determination  for  some  subset  of 
the  MOEswtll  likely  result  In  an  overall  "go"  for  that  general  objective.  AFHRL  SMEs  are  In  the  best 
position  to  determine  which  MOEs,  or  combinations  of  MOEs,  are  required  to  pass  a  given  Readiness 
test.  After  studying  the  MOEs  for  each  general  objective,  the  SMEs  should  recommend  the  minimum  set 
of  MOEs  that  must  be  assessed  a  "go"  condition.  If  those  MOEs  are  not  passed,  then  that  particular  Read¬ 
iness  test  will  have  failed.  The  software  that  falls  to  perform  the  critical  functions  should  be  fixed  be¬ 
fore  the  AOTS  is  released  to  the  workcenters  for  SLT&E. 
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-4.3  Test  Procedures 
-4.3. 1  Description 

Functionally  related  testing  objectives  will  be  assembled  into  procedures  that  are  logically  accomplished 
together. 

Test  data  values  are  required  for  testing.  Exhaustive  testing  of  ail  possible  data  values  is  not  feasible  be¬ 
cause  of  the  time  and  resources  involved.  However,  representative  test  esses  will  be  carefully  selected 
from  the  set  of  all  possible  values.  For  example,  if  a  menu  has  choices  a,  b,  c,  and  d.  then  the  test  cases 
may  include  a,  b,  e,  d,  e  ( undefined  alpha),  ( blank),  2  ( numeric),  and  ab  ( undefined  alpha).  Often  a 
view  of  the  code  will  offer  other  test  case  opportunities.  Loop  and  decision  constructs  can  be  tested  at 
their  boundary  conditions.  Test  case  selection  is  hampered  by  8  lack  of  a  data  dictionary  that  list  allowa¬ 
ble  values,  and  synonyms. 

-4.3.2  Purpose  of  Doatlled  Test  Procedures 

Test  procedures  will  record  and  prescribe  standard  test  conditions.  Required  hardware  and  software  will 
be  described.  The  procedures  will  contain  step-by-step  Instructions  and  expected  responses.  Instruc¬ 
tions  will  be  written  in  the  active  voice  and  begin  with  an  action  verb.  The  goal  of  the  procedures  when 
accomplished  is  to  result  in  a  written  historical  record  of  the  test  to  the  detail  necessary  for  future  eval¬ 
uators  to  reproduce  the  test  and  results. 

4.4  Tost  Execution 

4.4.1  Description 

The  testing  will  be  accomplished  by  members  of  the  Instructional  Systems  Development  Team.  These 
team  members  are  AF  specialty  area  experts,  and  are  considered  proficient  in  the  operation  of  AOTS  due  to 
their  extended  association  with  the  program  as  curriculum  developers. 

4.4.2  Responsibilities 

4.4.2. 1  BSED/BDM 

As  a  follow-on  from  the  AF-epproved  Readiness  Test  Plan,  BSED/BDM  will  write  test  procedures  to  guide 
theAF  test  team.  BSED/BDM  will  brief  the  AF  test  cadre  on  the  purpose  and  use  of  the  test  procedures. 
During  test  performance,  BSED/BDM  will  evaluate  the  AF  test  teem  performance  to  improve  quality. 

4. 4.2.2  AFHRL 

AFHRL  will  Provide  overall  management  to  the  Readiness  Test  program.  The  AFHRL  will  assure  that  each 
participating  organization  is  coordinated  in  action  and  integrate  their  respective  efforts  into  a  single 
group  effort.  AFHRL  personnel  will  perform  the  test  procedures. 

4. 4.2.3  DAC 

DAC  must  make  the  AOTS  prototype  available  in  time  to  support  the  Readiness  Test  schedule.  They  will 
provide  timely  analysis  of  apparent  testing  failures.  DAC  will  correct  correct  testing  failures  and  docu¬ 
ment  code  corrections.  DAC  will  perform  their  responsibilities  in  time  to  support  the  testing  schedule. 

4.5  Data  Recording,  Reduction,  Reporting 
4.5. 1  Description 

Measured  test  values  will  be  integrated  into  the  test  procedure  document  Data  will  be  recorded  on  the 
test  procedures  in  the  area  provided  for  that  purpose.  In  addition  to  required  data,  expert  testers  and  ob¬ 
servers  will  record  subjective  comments  and  other  pertinent  information  on  the  completed  procedures. 
Apparent  abnormalities  in  the  test  data  will  be  documented  on  SPRs  which  will  be  used  to  track  and  sus¬ 
pense  the  areas  of  concern. 

The  original  test  procedures  and  recorded  data  sheets  along  with  espies  of  the  SPRs  will  be  Included  with 
the  final  report. 
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4.5.2  ft*speftsto111t1«9 

4.5.2. 1  AFHRL 

The  greeter  pari  of  the  data  recording  and  analysts  tasks  will  be  performed  by  the  AF.  The  AF  will 
tain  the  successful  completion  of  each  test  procedure.  At  the  end  of  each  testing  day,  the  AF  may  brief  t 
responsible  organizations  on  their  findings.  The  AF  will  establish  a  schedule  for  the  timely  analysis  of 
test  abnormalities  which  are  recorded  on  SPRs. 

4.5.2.2  SSED/BDM 

The  role  of  the  BSED/BOM  team  has  been  to  design  a  Readiness  Test  around  the  restraints  established 
the  AF.  While  the  BSSD/8DM  team  has  no  direct  role  In  date  recording,  reduction,  or  analysis,  the  tee 
will  be  available  on  an  "on  call"  basis  to  advise  AFHRL  on  technical  matters. 

4.5.2.3  DAC 

DAC  is  responsible  for  the  performance  of  the  system.  When  test  data  shows  apparent  abnormalities,  1 
incumbent  on  the  DAC  to  analyse  these  abnormalities.  DAC  will  process  in  a  timely  manner  the  SPRs  t 
are  generated  during  testing. 
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5.0  Resources 

5.)  Facilities 

Facilities  requirements  are  identified  in  the  DAC  AOTS  Facility  and  Installation  Plan. 

5.2  Personnel 

Three  Air  Force  testers;  BALL/BDM  observers,  as  required;  DAC  software  engineers,  as  required. 

5.3  Test  Environment 

5.3. 1  Hardware 

The  testers  will  use  Z-248  microcomputer  workstations,  Alps  model  2000  printers,  and  Scantron  opti¬ 
cal  readers  to  perform  the  WORK  STATION  CHECK-OUT  procedures.  All  other  Readiness  Test  procedures 
will  be  accomplished  using  up  to  three  sets  of  representative  equipment. 

5.3.2  Software 

It  is  anticipated  that  AOTS  Release  1 .9  will  contain  most  of  the  software  required  to  assess  the  critical 
functions  described  in  this  Readiness  Test  Plan.  Release  1 .3  allows  users  to  enter/edit  a  Master  Task  List 
(MTL),  Generic  Position  Training  Requirements  (GPTRs),  Operational  Position  Training  Requirments 
(OPTRs),  and  Airmen  Training  Records  (ATRs).  However,  the  critical  function  of  generating  Individual 
Training  Requirements  ( ITRs)  is  not  expected  until  Release  1.9.  In  addition,  the  on-line  test  presenta¬ 
tion  and  off- line  test  generation  capabilities  will  not  be  available  until  Release  2.0.  This  incremental 
release  of  software  to  perform  AOTS  critical  functions  may  dictate  an  "incremental"  readiness  test. 

5.3.3  Data  Bases 

Data  to  conduct  the  1 4  Readiness  tests  described  In  Section  4  is  a  critical  resource.  If  real  data  Is  not 
available,  "dummy"  GPTRs  will  be  required  to  assess  the  capability  to  generate  an  OPTR.  Similarly, 
"dummy"  ATRs  will  be  required  to  evaluate  the  capability  of  the  system  to  generate  the  ITR  from  the  OPTR 
-  and  several  ATRs.  Detailed  descriptions  of  the  required  data  for  the  Readiness  Tests  will  be  added  to  this 
Test  Plan  in  the  next  few  weeks  as  the  final  versions  of  the  software  emerge. 
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6.0  TEST  SCHEDULE 

The  following  schedule  refers  to  the  delivery  ( "D"  day)  of  the  completed  AOTS  software.  The  days  after 
delivery  of  the  software  (e.g. ,  D  ♦  5)  indicate  work  days.  In  reelity,  all  of  the  software  or  required  test 
data  is  not  likely  to  be  read/  at  the  same  time.  This  Issue  contributed  to  the  overall  design  as  several 
separate  tssts  with  stand  alone  test  procedures.  However,  the  recommended  schedule  depicted  below  in¬ 
cludes  the  prefered 
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FIGURE  4-1.  ROTS  READINESS  TESTING  FUNCTIONAL  FLOti)  DIAGRAM 
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FIGURE  4-1  (Continued) 
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Flowchart  is  continued  on  pages  24  and  25. 
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FIGURE  4-1  (Continued) 
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FIGURE  4-1  (Concluded) 


